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Abstract 


Several possible statistical interpretations of the null result of the KARMEN 2 
neutrino oscillation experiment are discussed with the aim of clarifying the 
implications of the fact that KARMEN 2 did not observe any of the expected 
background events. The formalism that allows to take into account the er¬ 
ror of the expected mean background in a Poisson process with background 
is presented and applied to the statistical analysis of the KARMEN 2 null 
result. The possibility of ignoring the expected mean background calculated 
for the KARMEN 2 experiment is discussed and it is shown that the result¬ 
ing exclusion curves are ultra-conservative, but may be a safe choice in such 
a controversial case. It is also shown that the sensitivity curves of neutrino 
oscillation experiments cannot be considered as exclusion curves. The impli¬ 
cations of the different statistical analyses of the KARMEN 2 null result for 
the compatibility of the results of the KARMEN 2 and LSND experiments is 
discussed. 
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I. INTRODUCTION 


The KARMEN collaboration has reported recently []^ a null result of the KARMEN 2 
experiment in the search for neutrino oscillations in the channel v^. This result 

is very interesting because the KARMEN neutrino oscillation experiment is sensitive to 
the same region in the plane of the neutrino mixing parameters sin^ 2Q and Am^ as the 
LSND experiment IQ whose results provide an evidence in favor of v^^ —^ Vf. and v^^ —>■ Vf. 
oscillations. Hence, the statistical interpretation of the KARMEN 2 null result (as well as 
that of the LSND result) is crucial in order to obtain an indication on the compatibility or 
incompatibility of the different results of the two experiments. 

The KARMEN experiment is located at at the ISIS spallation neutron facility of the 
Rutherford Laboratories (UK). The proton beam produced at ISIS is pulsed in time: two 
100 ns wide proton pulses separated by 330 ns are produced every 20 ms. The proton pulses 
are directed on a target where they produce positive pions (the negative pions are absorbed 
in the source before decaying) which decay at rest according to the process 

7r+ ^ 

i (1-1) 

e+ + z/e + , 

producing an equal number of z/^, v^. and P^. The KARMEN experiment searches for Pg 
produced by P^ —Pg oscillations at a mean distance of 17.6 m. The time structure of the 
neutrino beam is important for the identihcation of the neutrino induced reactions in the 
KARMEN detector and for the effective suppression of the cosmic ray background. The 
KARMEN experiment started in 1990 and run until 1995 as KARMEN 1 0. In 1996 it was 
upgraded to KARMEN 2, eliminating the main cosmic ray induced background component 
in the search for P^ —>■ Pg oscillations [^]. 

So far the KARMEN 2 experiment measured no events [Q], with an expected background 
of 2.88 ± 0.13 events. This null result has been analyzed with the following statistical 
methods: 

Bayesian Approach. This method is recommended by the Particle Data Group and 
has been used by the KARMEN collaboration [^. The resulting upper limit for the 
mean /i of true neutrino oscillation events is 2.3 and the corresponding exclusion curve 
is reproduced in Fig. [| (the solid curve passing through the hlled squares). 

Unified Approach. This method has been proposed recently by Feldman and Cousins Q 
and has already been adopted by the Particle Data Group ^ as the new statistical 
standard for the calculation of frequentist conhdence intervals and upper limits with 
the correct coverage. The Unihed Approach is very attractive because it allows to 
construct classical conhdence belts which “unify the treatment of upper conhdence 
limits for null results and two-sided conhdence intervals for non-null results”. On the 
other hand, the Unihed Approach has the undesirable feature that when the number of 
observed events is smaller than the expected background, the upper limits for the mean 
/X of true neutrino oscillation events decreases rapidly when the background increases. 
Hence, by observing less events than the expected background an experiment can 
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establish a very stringent upper bound on /i even if it is not sensitive to such small 
values of /i. This is what happens in the Unihed Approach analysis of the KARMEN 2 
null result: the upper limit for the mean /i of true neutrino oscillation events is 1.1 
and the corresponding exclusion curve |ll] reproduced in Fig. [| (the solid curve passing 
through the hlled circles) is very stringent, even if the KARMEN 2 experiment is not 
sensitive to the excluded region of the neutrino oscillation parameters (see Ref. P|). 


New Ordering Approach. This recently proposed method is based on a new ordering 
principle for the construction of the classical conhdence belt which has all the desirable 
properties of the one calculated with the Unihed Approach and in addition minimizes 
the effect on the resulting conhdence intervals of the observation of less background 
events than expected. Hence, it is appropriate for the statistical interpretation of the 
null result of the KARMEN 2 experiment. The resulting upper limit for the mean 
/i of true neutrino oscillation events in the KARMEN 2 experiment is 1.9 and the 

is reproduced in Fig. |l| (the solid curve passing 


corresponding exclusion curve 
through the hlled triangles). 
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The Unihed Approach and the New Ordering Approach are reviewed in Section |I|. From 
Fig. 1^ one can see that the Bayesian Approach and the Unihed Approach yield rather 
diherent exclusion from the null result of the KARMEN 2 experiment. This diherence is 
crucial in order to assess the compatibility of the KARMEN 2 null result with the positive 
result of the LSND experiment, whose 90% CL allowed region ^ is shown as the shadowed 
area in Fig. |I]. The bayesian exclusion curve of KARMEN 2 is compatible with a large 
part of the LSND-allowed region, whereas the exclusion curve obtained with the Unihed 
Approach is incompatible with almost all the LSND-allowed region. The exclusion curve 
obtained with the New Ordering Approach lies close to the bayesian exclusion curve and 
tends to support the compatibility of the KARMEN 2 and LSND results. This is a desirable 
achievement. Furthermore, it is important to emphasize that, as explained in Section]^, the 
New Ordering Approach gives a correct frequentist coverage as the Unified Approach. 

In view of the uncertainty of the physical meaning of the KARMEN 2 null result induced 
by the signihcant diherence between the exclusion curve obtained in the Unihed Approach 
on one hand and the exclusion curves obtained with the Bayesian Approach and with the 
New Ordering Approach on the other hand, it is interesting to explore other possibilities for 
the statistical interpretation of the KARMEN 2 null result. 

In Sections EYHV|i discuss the following alternative statistical interpretations of the null 
result of the KARMEN 2 experiment, which take into account the problem of the non¬ 
observation of expected background events in the KARMEN 2 experiment: 


Larger Background Error. Since no background events have been measured in the KAR¬ 
MEN 2 experiment, it is possible to doubt of the correctness of the calculated error for 
the mean expected background, b = 2.88±0.13. Hence, one can consider the possibility 
that the error has been underestimated, for example, by an order of magnitude. This 
approach is discussed in Section B together with a presentation of the appropriate 
formalism. 


Unknown Background. An extreme attitude towards the doubting of the correctness of 
the calculated mean expected background b = 2.88 ±0.13 is to ignore it and to assume 
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that the background is unknown. In this case the KARMEN 2 null measurement gives 
an estimation of the mean /i + 6 of signal plus background events. This approach is 
discussed in Section 

Sensitivity. The sensitivity of an experiment is dehned as “the average upper limit that 
would be obtained by an ensemble of experiments with the expected background and 
no true signal” Q . The sensitivities of the KARMEN 2 experiment calculated with the 
Unihed Approach and with the New Ordering Approach are shown in Fig. (the solid 
curves passing through the empty circles and triangles, respectively). One could think 
of considering the sensitivity curve of the KARMEN 2 experiment as its exclusion 
curve. This approach is discussed and shown to be incorrect in Section 0. 


II. POISSON PROCESSES WITH BACKGROUND 

The events measured in the KARMEN experiment follow a Poisson distribution. The 
probability to observe a number n of events in a Poisson process consisting in signal events 
with mean p and background events with known mean h is 

P(n|p; 6 ) = 1 + ( 2 . 1 ) 

n! 

The classical frequentist method for obtaining the conhdence interval for the unknown pa¬ 
rameter /i is based on Neyman’s method to construct a confidence belt. This conhdence belt 
is the region in the n-/i plane lying between the two curves ni(/i; b, a) and ri 2 (/i; b, a) such 
that for each value of /i 


n2{fJ,-,b,a) 

F(n G [ni(/i; 6, a), U2(p; 6, a)] Ip; 6) = P(n|p;6) = a, (2.2) 

n=ni 

where a is the desired conhdence level. The two curves ni{fi;h,a) and n 2 (p; 6 , a) are re¬ 
quired to be monotonic functions of and can be inverted to yield the corresponding curves 
pi(n; h, a) and iJ.2{n; b, a). Then, if a number Uobs of events is measured, the conhdence inter¬ 
val for fi is [p 2 (?^obs; b, a), pi(uobs; b, a)]. This method guarantees by construction the eorrect 
coverage, i.e. the fact that the resulting conhdence interval [p 2 (^obs; a), hi(?^obs; &, «)] is a 

member of a set of conhdence intervals obtained with an ensemble of similar experiments 
that contain the true value of p with a probability a. Actually, in the case of a Poisson 
process, since n is an integer, the relation o can only be approximately satished and 
in practice the chosen aeceptance intervals [ni(p; 6 , a), U 2 (p; 6 , a)] are the smallest intervals 
such that 


P{n e [? 7 ,i(p; 6 , a), ? 7 , 2 (p; &, a)]|p; 6 ) > a. (2.3) 

This choice introduces an overcoverage for some values of p and the resulting conhdence 
intervals are conservative. As emphasized in Ref. conservativeness is an undesirable 
but unavoidable property of the conhdence intervals in the case of a Poisson process (it 
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is undesirable because it implies a loss of power in restricting the allowed range for the 
parameter p). 

The construction of Neyman’s conhdence belt is not unique, because in general there 
are many different couples of curves ni(p; b, a) and n 2 (/i; b, a) that satisfy the relation ( p.2D . 
Hence, an additional criterion is needed in order to dehne uniquely the acceptance intervals 
[ni(/i; b, a), 77-2(p; b, a)]. The two common choices are 

1 — a 

P{n < ni(p; b, a)|/i; b) = P{n > n 2 (/i; b, a)|/i; b) = —-— , (2.4) 

which leads to central eonfidence intervals and 

P{n < 77-1 (p; b, a)\qi;b) = 1 — a , (2.5) 


which leads to upper confidence limits. Central conhdence intervals are appropriate for 
the statistical description of the results of experiments reporting a positive result, i.e. the 
measurement of a number of events signihcantly larger than the expected background. On 
the other hand, upper conhdence limits are appropriate for the statistical description of 
the results of experiments reporting a negative result, i.e. the measurement of a number of 
events compatible with the expected background. However, Feldman and Cousins ^ noticed 
that switching from central conhdence level to upper conhdence limits or vice-versa on the 
basis of the experimental data (“hip-hopping”) leads to undercoverage for some values of p, 
which is a serious haw for a frequentist method. 

Feldman and Cousins [Q proposed an ordering principle for the construction of the 
acceptance intervals that is based on likelihood ratios and produces an automatic transition 
from central conhdence intervals to upper limits when the number of observed events in a 
Poisson process with background is of the same order or less than the expected background, 
guaranteeing the correct frequentist coverage for all values of p. The acceptance interval for 
each value of p, is calculated assigning at each value of 77 a rank obtained from the relative 
size of the ratio 


RuA{n\p;b) 


P{n\p;b) 
P(77,|/7best j b'j 


( 2 . 6 ) 


where /Xbest = Phest{n;b) (for a hxed b) is the non-negative value of p that maximizes the 
probability P{n\p-,b)\ 


Phesfin-, b) = max[0, n-b]. (2.7) 

As emphasized in Ref. 0, “i? is a ratio of two likelihoods: the likelihood of obtaining n 
given the actual mean p, and the likelihood of obtaining n given the best-ht physically 
allowed mean”. For each hxed value of p, the rank of each value of n is assigned in order of 
decreasing value of the ratio R\]A{n\p-,b)\ the value of n which has bigger R\jA{n\p-,b) has 
ra nk one, the value of n among the remaining ones which has bigger R\]A{n\p-,b) has rank 
two and so on. The acceptance interval for each value of p is calculated by adding the values 
of 77 in increasing order of rank until the condition (|2.3| ) is satished. 

The automatic transition from two-sided conhdence intervals to upper conhdence limits 
for 77 < & is guaranteed in the Unihed Approach by the fact that pbest is always non-negative. 
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Indeed, since ^hest{n<b-, b) = 0, the rank of n < 6 for /i = 0 is one, implying that the interval 
0 < n < b for /x = 0 is guaranteed to he in the conhdence belt. 

Although the Unihed Approach solves brilliantly the problem of obtaining a transition 
with correct coverage from two-sided conhdence intervals to upper conhdence limits for 
n <b, it has the undesirable feature that when n <b the upper bound /ii(n; b, a) decreases 
rather rapidly when b increases and stabilizes around a value close to 0.8 for large values of b. 
From a physical point of view this is rather disturbing, because a stringent upper bound for /i 
obtained by an experiment which has observed a number of events significantly smaller than 
the expected background is not due to the fact that the experiment is very sensitive to small 
values of n, but to the fact that less background events than expected have been observed. 

This is the case of the null result of the KARMEN 2 experiment, from which the Unihed 
Approach yields an upper 90% CL conhdence limit for the mean /x of neutrino oscillation 
events of 1.1 events. The corresponding exclusion curve in the plane of the neutrino mix¬ 
ing parameters sin^ 26 and Am^ is shown in Fig. (the solid curve passing through the 
hlled circles) and one can see that it is signihcantly more stringent than the exclusion curve 
obtained with the Bayesian Approach (the solid curve passing through the hlled squares). 
The strictness of the Unihed Approach exclusion curve is due to the non-observation of the 
expected background events and not to the sensitivity of the experiment (see the discussion 
T0[| and the sensitivity of the KARMEN experiment presented in Ref. 


in Ref. 


This 


is clearly an undesirable result from a physical point of view, because the statistical inter¬ 
pretation of the data produces an exaggeratedly stringent result that could lead to incorrect 
physical conclusions. 

The discrepancy between the Bayesian Approach exclusion curve and the Unihed Ap¬ 
proach exclusion curve is worrying for a physicist, because the Bayesian Approach exclusion 
curve is compatible with a large part of the LSND-allowed region (the shadowed area in 
Fig. i, whereas the Unihed Approach exclusion curve excludes almost all the LSND al¬ 
lowed region. 

In Ref. ^ 


10] I have proposed a new ordering principle for the construction of a classical 


conhdence belt which has all the desirable features of the one in the Unihed Approach {i.e. 
an automatic transition with the correct coverage from two-sided conhdence intervals to 
upper conhdence limits when the observed number of events is of the order or less than the 
expected background) and in addition minimizes the decrease of the upper conhdence limit 
for a given n as the mean expected background b increases. The new ordering principle is 
implemented as the Feldman and Cousins ordering principle in the Unihed Approach, but 
for each value of /x the rank of each value of n is calculated from the relative size of the 
likelihood ratio 

P(n|/x;6) 


RNo{n\p.] b) = 


( 2 . 8 ) 


F(n|/Xref; b) ’ 

where the reference value /Xref = fiveiin; b) is taken to be the bayesian expected value for /x: 


href(n; b) = 


\fc =0 ■ / \fc =0 / 


-1 


(2,9) 


Here P(/i|n; b) is the bayesian probability distribution for /x calculated assuming a constant 
prior for /x > 0 (see, for example, ||11||): 
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-1 


F(/i|n; h) = {h + ^iYe ^ ^ 


fc =0 


( 2 . 10 ) 


The obvious inequality X]fc=o ™plies that fi^ei{n;b) > 1. Therefore, 

firef{n] b) represents a reference value for /i that not only is non-negative, as desired in order 
to have an automatic transition from two-sided intervals to upper limits, but is even bigger 
or equal than one. This is a desirable characteristics in order to obtain a weak decrease of 
the upper confidence limit for a given n when the expected background b increases. Indeed, 
it has been shown in Ref. that for n < 6 the upper bound fii{n;b,a) decreases rather 


weakly when b increases and stabilizes around a value close to 1.7 for large values of b. This 
behaviour of fii{n]b,a) is more acceptable for the physical interpretation of experimental 
results than the behaviour of b, a) in the Unified Approach. 

The 90% CL upper confidence limit for the mean /i of neutrino oscillation events following 
from the analysis of the KARMEN 2 null result with the New Ordering Approach is of 1.9 
events and the corresponding exclusion curve in the plane of the neutrino mixing parameters 
sin^ 20 and Am^ is shown in Fig. |l] (the solid curve passing through the hlled triangles). 
One can see that this exclusion curve, although obtained with a method that guarantees the 
correct frequentist coverage as the Unified Approach, is significantly less stringent than the 
one obtained with the Unified Approach and lies close to the bayesian exclusion curve. 

Hence, the New Ordering Approach has solved the apparent conflict between the frequen¬ 
tist and bayesian statistical interpretation of the null result of the KARMEN 2 experiment: 
by choosing an appropriate ordering principle in the construction of the eonfidence belt, the 
exclusion curve obtained with the freguentist method is in reasonable agreement with the one 
obtained with the Bayesian Approach. 

Nevertheless, some concern still remain on the interpretation of the fact that the KAR¬ 
MEN 2 experiment did not observe any of the expected background events. Such concern will 
increase in the future if the KAREN 2 experiment will continue to observe significantly less 
events than those expected from the background. Hence, it is interesting to explore other 
possibilities for the statistical interpretation of the result of the KARMEN2 experiment. 
This is the aim of the following three Sections. 


III. LARGER BACKGROUND ERROR 

Since no background events have been measured in the KARMEN 2 experiment and the 
mean expected background is b = 2.88 ±0.13, it is possible to doubt of the correctness of the 
calculated error for the mean expected background. In this Section, I will investigate which 
would be the physical implications of the KARMEN 2 null result if the calculated error for 
the mean expected background has been underestimated by an order of magnitude, i.e. I 
will take b = 2.88 ± 1.3. I will assume a normal probability distribution function for the 
mean expected background b: 


b, ab) 



{b-by 


(3.1) 
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with b = 2.88 and at = 1.3. For simplicity, since at ^b/3, I will consider b varying from 
—oo to +CXD, neglecting the small error introdnced by considering negative valnes of b. This 
approximation allows an analytic solution of the integrals involved in the calculation[|. 

If p. is the mean of true signal events, the probability P{n\fi]b, ab) to observe n events is 
given by 

F(n|p;6, (Tb) = j P(n|p;6) f{b;b,ab) db, (3.2) 

with the Poisson probability P(n|p; b) given in Eq.( p.l|) . Hence, P(n|p; 6, ab) can be written 
as 


P(n\fi;b,ab) = ^ ^ 

n\ ^^ 

k=0 


1) ^P-^h{\ab) 


with 


Ik{b,ab) = 




/•+00 

r. ib-W] 

/ b exp 

/ —00 

L 2^2 J 


db. 


The integral Ik{b, ab) can be written as 


h{b,ab) = exp 


2 


E 


{b - alf ’ aimj, 


(3.3) 


(3.4) 


(3.5) 


where m, is the central moment of the normal distribntion with unit variance, 

-I r+oo 

nij = _ / V db. (3.6) 

Taking into account that jbe~^^/‘^db = —the integral in Eq. (|3.6|) can be calculated 
by parts, yielding 


TTlj = 


(j/2)! 2P2 


(3.7) 


for j even and rrij = 0 for j odd. 
From Eqs.l 


and (|3.7|), for the probability of n events one obtains 


^If b is restricted to the interval [0,+ 00 ) the exact normalization factor of f{b;b,ab) replacing 
IjV^ab in Eq.(^) is 1/N(b,ab) with 


Nib,ab) = 


1 + erf 


(v/2<rj 


CTfe, 


In this case N{b,ab) and the integral in Eq.(p.2p must be calculated numerically. 
















(3.8) 


with 


F(n|/i; b, at) = exp 


■(A + 5) + y 


A:=0 


^ ..n—k 

H' 


{n — k)\ 


Jkip) ^b) y 


fc /2 

Jk(b,ab) = 

i=o 


{k-2j)\j\2^ 


(3.9) 


Equation ( p.8|) gives the formula for the probability P{n\^;b,ab) to observe a number 
n of events in a Poisson process consisting in signal events with mean /x and background 
events with known mean 6 = 6 ± cTf,, i.e. it replaces Eq. (p.l|) if the error ab of the calculated 
mean background is not negligible (but ab < b/3). The construction of the confidence 
belt follows the same procedure described in Section |1| and now the confidence interval for 
/i corresponding to a number riobs of observed events is [/i 2 (’^obs; 5, cxft, a),/xi(nobs; Cfe, a)], 
i.e. it depends on b and ab. The acceptance intervals can be constructed following the 
same principles discussed in Section i.e. one can construct the confidence belt for central 
conhdence intervals or upper confidence limits, or the confidence belt in the Unified Approach 
or in the New Ordering Approach. In the remaining part of this Section I will present 
the formalism for the implementation of the Unified Approach and of the New Ordering 
Approach and I will discuss the corresponding implications for the statistical interpretation 
of the KARMEN 2 null result. 

The quantity fx\:,est{n;b,ab) in the Unified Approach is the value of fi that maximizes 
F(n|/x; 6, (Tb) and the acceptance interval for each value of fi is calculated assigning at each 
value of n a rank obtained from the relative size of the ratio 


R\jA{n\fi;b, ab) 


F(n|/x;6, ab) 
^(^lhbest;5, ab) 


(3.10) 


The value of /Xbest(’^; b, ab) can be easily calculated by hand for n = 0,1, 2, whereas for higher 
values of n it can be calculated numerically. The resulting confidence belt for b = 2.88 and 
ab = 1.3 is plotted in Fig. ^ (the region between the two dashed lines), where it is compared 
with the confidence belt for ab = 0 (the region between the two solid lines), which practically 
coincides with the confidence belt for ab = 0.13. The confidence belt for ab = 1.3 is larger 
than the one for ab = 0 because the integral in Eq. (p.2|) has the effect of flattening the 
probability F(n|/x; 6, ab) as a function of n for fixed /x with respect to F(n|/x; b = b) and this 
flattening effect increases with the size of ab. 

From Fig. ^ one can see that the upper limit /xi for n = 0 is of 1.2 events, only slightly 
larger than the upper limit of 1.0 events obtained for ab = 0. Therefore, the exclusion curve 
that follow from the null result of the KARMEN 2 experiment if ab = 1.3 lies close to the 
solid curve passing through the hlled circles in Fig. (corresponding to /xi = 1.1)0. 


^In the Unified Approach of Ref. Cfe = 0 but the upper limit /xi for n = 0 is of 1.1 events 
because /xi is forced to be a non-increasing function of h. 
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In the New Ordering Approach the acceptance interval for each value of /i is calculated 
assigning at each value of n a rank obtained from the relative size of the ratio 


RKo{n\^;b, ab) 


ab) 

P{n\fireub, ab) ’ 


(3.11) 


where the reference value /iref = Hreiin^b, ab) is the bayesian expected value for /i. The value 
of firei{,n]b,ab) can be calculated analytically. For the bayesian probability distribution 
function for /i with a constant prior, 


one obtains 


P{fi\n;b,ab) 


P{n\^;b, ab) 
P{n\n-,b,ab) d/i ’ 


P{fi\n;b,ab) = e ^ 




Jk{b, ab) 


-1 


^ Jfc(6, ab) 


, fc =0 


(3.12) 


(3.13) 


Hence, the reference value fireiin] b, ab), which is the bayesian expected value for /x, is given 
by 


^b) 


-1 


n + 1 


Jk{b,ab) y^^Jkjb, ab) 


K k = 0 


,k=0 


(3.14) 


The resulting conhdence belt in the New Ordering Approach for b = 2.88 and ab = 1.3 
is shown in Fig. ^ (the region between the two dashed lines), where it is compared with 
the conhdence belt for ab = 0 (the region between the two solid lines), which practically 
coincides with the conhdence belt for at = 0.13. One can see that the upper limit for /i if 
’^obs = 0 is of 2.0 events, only slightly larger than the upper limit of 1.9 events obtained for 
ab = 0. Therefore, the KARMEN 2 exclusion curve for ab = 1.3 lies close to the solid curve 
passing through the hlled triangles in Fig. (for ab = 1.3 the ordinate of the exclusion curve 
for high Am^’s is sin^ 29 = 2.5 x 10“^, compared with sin^ 29 = 2.3 x 10“^ for at = 0). 

The conclusion that one can reach from the results presented in this Section is that the 
error of the calculated mean background has little ehect on the exclusion curves obtained 
in the Unihed Approach and in the New Ordering Approach (at least for ab < b/3). Even 
if the error ab of the mean background in the KARMEN 2 experiment is increased by an 
order of magnitude with respect to the calculated one {ab = 0.13), the exclusion curves in 
the Unihed Approach and in the New Ordering Approach are practically equivalent to the 
corresponding curves calculated for ab = 0. 


IV. UNKNOWN BACKGROUND 

Since the KARMEN 2 experiment did not observe any event, with a mean expected back¬ 
ground b = 2.88 ± 0.13, and even increasing by an order of magnitude the estimated error of 
the mean expected background does not help in solving the discrepancy among the exclusion 
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curves calculated with different statistical methods, one can consider the extreme possibility 
of doubting of the correctness of the calculated mean expected background and ignore it. 
In other words, one can assume that the background is unknown and that the KARMEN 2 
null measurement gives an estimation of the mean /i + 6 of signal plus background events. 

This approach leads to an upper confidence limit (/r + 6)i which depends only on the 
observed number of events. Since & > 0, the upper confidence limit (p + 6)i for fi + b is 
also a conservative upper confidence limit for p. Although the derivation of a conservative 
upper confidence limit is generally an undesirable result, in controversial cases as that of the 
KARMEN 2 experiment it may be a safe choice. 

Actually, in the implementation of the Unknown Background approach one has also 
to chose the method for building the confidence belt. I will consider the following four 
possibilities, whose confidence belts are plotted in Fig. 

Standard confidence belt for upper confidence limits. The resulting 90% CL confi¬ 
dence belt is the region below the solid line in Fig. |^. This method implies an upper 
90% CL confidence limit of 2.3 events forfi + b if the number of observed events Uohs 
is zero. Taking it as a conservative upper confidence limit for /i leads to an exclusion 
curve coinciding with the bayesian exclusion curve shown in Fig. ^ This exclusion 
curve is plotted in Fig. ^ (the solid curve passing through the filled squares). 

Standard confidence belt for central confidence intervals. The resulting 90% CL 
confidence belt is the region between the two dashed lines in Fig. ^ and the 90% 
CL upper limit for /i-|-6 is of 3.0 events if Uohs = 0. The corresponding exclusion curve 
is plotted in Fig. ^ (the solid curve passing through the dotted squares). 

Unified Approach. The 90% CL confidence belt is the region between the two dash- 
dotted lines in Fig. ^ and the upper limit for fi + b is of 2.4 events for riobs = 0. The 
corresponding exclusion curve is plotted in Fig. (the solid curve passing through the 
dotted circles). 

New Ordering Approach. The 90% CL confidence belt is the region between the two 
dash-dot-dotted lines in Fig. and implies an upper limit of 2.6 events for fi + b if 
nohs = 0. The corresponding exclusion curve is plotted in Fig. (the solid curve 
passing through the dotted triangles). 

One can see that the four exclusion curves in Fig. ^ obtained assuming an unknown 
background with the four different statistical methods listed above do not differ much. 
This is a good sign and in practice tends to support the exclusion curve obtained from 
the standard confidence belt for upper confidence limits, which coincides with the bayesian 
exclusion curve plotted in Fig. Q. 

However, it must be noted that the choice to use the Unknown Background approach 
has been done on the basis of the result of the experiment. This procedure introduces some 
overcoverage, as in the flip-hopping policy discussed by Feldman and Cousins 0. 

For example, one can consider the possibility to take b = nobs for nobs < 3 and b = 2.88 for 
nobs > 3. Using the standard confidence belts for upper confidence limits for 6 = 0,1, 2, 2.88 
shown in Fig.^ one obtain the confidence belt below the thick solid line in Fig.^. It is clear 
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from the figure that this policy leads to overcoverage for /i < 3.3, that is the interesting 
region in practice. The value of the induced overcoverage is shown in Fig.|^ where I plotted 
the sum of the probabilities corresponding to the n’s lying in the acceptance interval for a 
given /X of the conhdence belt below the thick solid line in Fig.^ as a function of /i. One 
can see that for /i < 3.3 this sum is even bigger than one! This is due to the fact that 
the probability has not been properly normalized to take into account that b depends on 
n 0. (The fact that for /x > 3.3 the sum of the probabilities is bigger than 0.90 shows the 
unavoidable overcoverage in the case of a Poisson processes due to the discreteness of n. 
See, for example. Refs. 0-i') 

In conclusion of this Section, we have seen that the Unknown Background approach 
is allowed and the corresponding exclusion curves obtained with four different statistical 
methods do not differ much. In practice this result represents a support for the bayesian 
exclusion curve plotted in Fig. |l|. However, it is necessary to emphasize that the Unknown 
Background approach is conservative for two reasons: 1) The upper conhdence limit for the 
mean /x neutrino oscillation events is taken to be equal to the upper conhdence limit for 
/i + 6, where b is the unknown mean background events; 2) The choice to use the Unknown 
Background approach is done on the basis of the result of the experiment, introducing some 
overcoverage. Hence, we can consider the exclusion curves obtained with the Unknown 
Background approach as ultra-conservative. 


V. SENSITIVITY 

In their Unihed Approach paper Feldman and Cousins suggested that in the cases in 
which the measurement is less than the estimated mean background, the experimental col¬ 
laboration should report also the sensitivity of the experiment, dehned as “the average upper 
limit that would be obtained by an ensemble of experiments with the expected background 
and no true signal”. This is also recommended by the Particle Data Group |P] . 

From the dehnition, the sensitivity /x^ of an experiment measuring a Poisson process with 
an expected mean background b with negligible error is given byQ 

OO 

/x^(6, a) = ^ /xi(?x; b, a) P(n|/x = 0, 6), (5.1) 

n=0 


^One can also calculate the coverage of the confidence belt below the thick solid line in Fig.^ for 
different fixed values of b. In this case one obtains overcoverage for small values of p if 6 is large 
(for example, p = 2 and b = 2.88) and undercoverage for large values of p if 6 is small (for example, 
p = 4 and 6 = 0). However, this calculation is in contradiction with the assumption that 6 is fixed 
by the procedure, i.e. b = n for n < 3 and 6 = 2.88 for n > 3. 

“^This formula can be easily generalized to the case of an expected mean background 6 = b ± ai, 
with non-negligible error a^: fJ.s(b, (Tb, a) = ®) P(^|p = 0; 6, Ub), with P(n|p; 6, ab) given 

by Eq.(|]^). 
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where a is the conhdence level of the upper limits The sensitivity /i^ depends 

on the values of the upper limits fii{n]b,a), which are different in the different methods 
for calculating the conhdence belt {e.g. in the Unihed Approach and in the New Ordering 
Approach). 

The sensitivity of the KARMEN 2 experiment for a = 90% is = 4.4 in the Unihed 
Approach and fig = 4.7 [|T^] in the New Ordering Approach. The corresponding sensitivity 
curves are shown in Fig. (the solid curves passing through the empty circles and triangles, 
respectively). 

One could think of considering the sensitivity curve of the KARMEN 2 experiment as its 
exclusion curve. However, this is wrong for the following reasons. From Eq.( b.l|) it is clear 
that fig is the expected value of the upper limit /ii under the hypothesis that /x = 0. This is 
the only correct meaning of pig Even though the value of pg depends on the conhdence level 
a, the interval [0,/is] does not have a conhdence level a from a frequentist point of view. 

Notice that fig does not depend on any experimental result and can be calculated for real 
as well as hypothetical experiments. If the sensitivity curve corresponding to fis{b,a) could 
be interpreted as an exclusion curve with conhdence level a, one could obtain exclusion 
curves {i.e. experimental results) without performing any experiment! This apparent para¬ 
dox is solved by noting that fig is not calculated by assuming that the number of observed 
events is zero (all values of n are taken into account in Eq.(^)) but assuming that the true 
value of p is zero and if the true value of /i would be known it would be useless to do any 
experiment to measure it. 

Each sensitivity curve in Fig. |I| is shifted to the right with respect to the corresponding 
exclusion curve, although it is calculated under the assumption that /x = 0, because the 
sensitivity curve takes into account the possibility that an experiment with mean background 
b can measure a number of events n > b, even if /x = 0. For example, in the Unihed 
Approach with b = 2.88 one has /xi(?x = 3;a = 0.90) = 4.5, /xi(?x = 4; a = 0.90) = 5.7, 
/ii(n = 5; a = 0.90) = 7.1, and so on. On the other hand, since /ii(n = 2; a = 0.90) = 3.1, an 
experiment measuring two events, in good agreement with the mean expected background 
b = 2.88, can draw an exclusion curve that is signihcantly more stringent than the sensitivity 
curve for b = 2.88. 

Since the sensitivity curve of a neutrino oscillation experiment can be calculated before 
doing the experiment, it is useful in order to plan future experiments that cover approx¬ 
imately the region of interest in the plane of the neutrino mixing parameters sin^ 29 and 
Am^. 


VI. CONCLUSIONS 


The statistical interpretation of the null result of the KARMEN 2 neutrino oscillation 
experiment is rather problematic because no events were observed with a mean expected 
background of 2.88 ± 0.13 events |l|. The exclusion curves obtained with the Bayesian Ap¬ 
proach and with the Unihed Approach are signihcantly diherent and yield contradicting 
indications on the compatibility of the KARMEN 2 result with the neutrino oscillation sig¬ 
nal measured in the LSND experiment [Q (see Fig. |^). The analysis of the KARMEN 2 
null result with the New Ordering Approach , which is a frequentist method with correct 
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coverage as the Unified Approach, yields an exclusion curve close to the one obtained with 
the Bayesian Approach. In this way, the undesirable discrepancy between frequentist and 
bayesian interpretations of the KARMEN 2 null result is removed. However, some concern 
still persist on the interpretation of the fact that none of the expected background events 
have been observed in the KARMEN 2 experiment. This concern will increase in the future 
if the KARMEN 2 experiment will continue to observe significantly less background events 
than expected. Hence, it is interesting to investigate other possibilities for the statistical 
interpretation of the KARMEN 2 null result. 

In this paper I have discussed three alternative statistical interpretation of the KAR¬ 
MEN 2 null result: the Larger Background Error possibility in Section B the Unknown 
Background possibility in Section |I^ and the Sensitivity possibility in Section 0. 

In Section Bi have presented the formalism that allows to take into account the error 
of the expected mean background. However, the conclusion of this section is that taking 
into account the error of the expected mean background in the KARMEN 2 experiment, 
even if it is wrong by an order of magnitude, does not help in solving the problem of the 
statistical interpretation of the result of this experiment, because the resulting exclusion 
curves are practically equivalent to the ones obtained assuming no error for the expected 
mean background. 

The Unknown Background approach discussed in Section Q gives ultra-conservative 
exclusion curves, which in the case of the KARMEN 2 experiment tend to support the ex¬ 
clusion curve obtained with the Bayesian Approach. Obtaining ultra-conservative exclusion 
curves is generally not desirable, but could be a safe choice in controversial cases as that of 
the KARMEN 2 experiment. 

The possibility to consider a Sensitivity curve as an exclusion curve is discussed in Sec¬ 
tion |V] and is shown to be wrong. Since the sensitivity curve of a neutrino oscillation 
experiment can be calculated before doing the experiment, its usefulness lies in the possibil¬ 
ity to plan future experiments in order to cover approximately the region of interest in the 
plane of the neutrino mixing parameters sin^ 29 and Am^. 

Finally, I would like to remark that the direct comparison of exclusion curves and al¬ 
lowed regions obtained with different statistical methods does not have a precise statistical 
significance. Hence, such a comparison cannot be used to combine the results of different 
experiments or to infer with some known confidence level a contradiction between the results 
of different experiments when the comparison is done on the border of the exclusion curves 
and of the allowed regions. Hence, the comparison of the KARMEN 2 exclusion curves 
and the LSND-allowed region, which was obtained with a different statistical analysis (see 
Ref. [^), must be done with great caution. In the future, if the KARMEN 2 experiment 
will continue to observe no neutrino oscillations, it will be possible to claim a contradiction 
between the results of the two experiments only when the exclusion curves obtained from 
the result of the KARMEN 2 experiment with different statistical methods will produce 
similar results and will lie well on the left of the region allowed by the results of the LSND 
experiment. 
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FIGURES 


Fig. i 90% CL exclusion curves in the plane of the neutrino oscillation parameters 
sin^20-Am^ corresponding to the null result of the KARMEN 2 experiment |^. The solid 
curves passing through the hlled squares, circles and triangles are obtained with the Bayesian 
Approach, the Unihed Approach and the New Ordering Approach, respectively. The solid 
curves passing through the empty circles and triangles are the sensitivity curves obtained 
with the Unihed Approach and the New Ordering Approach, respectively. The shadowed 
area is the region allowed at 90% CL by the results of the LSND experiment and the 
dashed, dash-dotted and dash-dot-dotted curves are the 90% CL exclusion curves of the 
Bugey ^ 


BNL E776 |T^ and CCER experiments, respectively. 


Fig. Conhdence belts for 90% CL obtained with the Unihed Approach for b = 2.88 
and at, = 0 (the region between the two solid lines) and ah = 1.3 (the region between the 
two dashed lines). 


Fig. |. Conhdence belts for 90% CL obtained with the New Ordering Approach for 
b = 2.88 and ah = 0 (the region between the two solid lines) and ah = 1.3 (the region 
between the two dashed lines). 


Fig. |. Conhdence belts for 90% CL obtained in the case of unknown background (see 
Section [TVl) with the standard method for upper conhdence limits (the region under the 
solid line), with the standard method for central conhdence intervals (the region between 
the two dashed lines), with the Unihed Approach (the region between the two dash-dotted 
lines) and with the New Ordering Approach (the region between the two dash-dot-dotted 
lines). The lower conhdence limits (/i -|- 6)2 calculated with the Unihed Approach and the 
New Ordering Approach coincide for n < 3. All the conhdence belts include the origin of 
the coordinates n, ^ + b. 


Fig. |. 90% CL exclusion curves obtained in the case of unknown background (see 
Section [TVl) with the standard method for upper conhdence limits (the solid curve passing 
through the hlled squares), with the standard method for central conhdence intervals (the 
solid curve passing through the dotted squares), with the Unihed Approach (the solid curve 
passing through the dotted circles) and with the New Ordering Approach (the solid curve 
passing through the dotted triangles). 


Fig. |. Standard 90% CL conhdence belts for upper conhdence limits for 6 = 0,1, 2, 2.88 
(the region below the dashed, dotted, dash-dotted and dash-dot-dotted line, respectively). 
The region below the thick solid line is the conhdence belt obtained taking b = n for n <3 
and b = 2.88 for n > 3. 
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Fig.0. Sum of the probabilities corresponding the n’s lying in the acceptance interval 
for a given ji of the conhdence belt below the thick solid line in Fig.^ as a function of /r. 
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